Experiments on determinism

• Author: Fang Zhang
• Date: 2016.6.15
• E-mail: fza34@sfu.ca

Experiment 20160525

1. Use fast tools to shuffle the sample fast file by using:

   fastq-sort -R —seed=3 input.fastq > output.fastq.
   

2. Run pipeline on shuffled fast files, very slow. Should check the order in the two fastq files.
3. Download the Beijing lineages.

Experiment 20160526

1. Run pipeline on shuffled samples and compare the results with unshuffled.

   nohup ./polyTB_Breezy_try.sh /home/zhf615/TB_test/MALAWI/sample /home/zhf615/TB_test/MALAWI/reference/AL123456.2 TBSequence ERR245648 /global/scratch/zhf615/MALAWI/data/fastq/ /home/zhf615/TB_test/MALAWI/input/RESI_SNPs/Resi-List-MasterV27_.vcf.gz &
   

2. Test Beijing lineage sequences.
3. Check the order of shuffled samples.

   cat  ERR245648_1_random.fastq | grep @ERR245648 | awk '{print $1}' | awk  -F '.' '{print $2}'  > ERR245648_1.txt
   cat  ERR245648_2_random.fastq | grep @ERR245648 | awk '{print $1}' | awk  -F '.' '{print $2}'  > ERR245648_2.txt
   

   Orders are not the same and have to use other shuffle methods.

Experiment 20160530

1. Use the shuffle code given by the paper.
2. terms:

• GATK = SNPs found with GATK
• MPILEUP = SNPS found by MPILEUP
• INTERSSECTION = intersection of both sets of SNPs
• dedup = duplicate mappinga are deleted
• shuffled = samples are shuffled
• lowQRM = low quality mappings are removed
• 17 = SNP with quality lower than 17 is removed

1. Compare the results between shuffled and unshuffled 3 samples, the results are in excel file below:

1. The results are not deterministic. Filter some low quality reads and base, GATK -nbq 30 filter 30, mpileup -Q 15, the 3 samples can have identical results in number and SNPs.

1. Try another 3 samples using the same parameters. They cannot have the same results which means GATK and mpileup may not be deterministic.

Experiment 20160604

1. As GATK and mpileup both are not deterministic (suppose BWA mem is deterministic), try freebayes.

1. All SNP callers are not deterministic, however, they all have the same results generated by the pipeline with the same parameters. Therefore, it seems that BWA is not deterministic that leads to undeterminism of GATK, mpileup and freebayes.

Experiment 20160608

1. mrFAST can have deterministic results on shuffled and unshuffled samples.
2. However it cannot work on fastq files whose reads have different lengths. Most samples in Beijing dataset have reads of different lengths.

3. use tool from Yanyi Lin to merge 2 fastq files and filter them by length.

   /home/zhf615/TB_test/MALAWI/tools/fastq_merge/extract -m 14 -s test_1.fastq -p test_2.fastq -o final.fastq
   cat final.fastq | paste - - - - - - - -|awk 'length($3)>=100 &&length($8)>=100 ' | sed 's/\t/\n/g'  > final.fastq
   /home/zhf615/TB_test/MALAWI/tools/fastq_merge/extract -m 17 -p final.fastq -o final
   

4. Because reads have different lengths, parameter —-crop 100 should be used in mrFAST command.